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Abstract 

The use of appropriate Educational Simulation systems (software and hardware for learning purposes) may 
contribute to the application of the "Learning by Doing" (LbD) paradigm in classroom, thus helping the students 
to assimilate the theoretical concepts of a subject and acquire certain pre-defined competencies in a more 
didactical way. The main objective of this work is to conduct a teaching experience using a flight simulation 
environment so that the students of Aeronautical Management degree can assume the role of an aircraft pilot, 
in order to allow the students understanding the basic processes of the air navigation and observe how the new 
technologies can transform and improve these processes. This is especially helpful in classroom to teach the 
contents of the Single European Sky ATM Research (SESAR) programme, a European project that introduces a 
new Air Traffic Management (ATM) paradigm based on several relevant technological and procedural changes 
that will affect the entire air transportation system in the short and medium term. During and after the 
execution of several activities with a flight simulator in the classroom, both the performance and satisfaction 
level of the students regarding the teaching experience have been assessed through four different assessment 
methods, i.e., objective test, execution test, observation techniques and attitude-opinion survey. 


Keywords - Teaching experience, Educational Simulation, Flight simulation, Learning by doing, Competencies. 


1 INTRODUCTION 

Educational Simulation (ES) is a term that refers to a variety of selectively representational and interactive 
environments that can represent or imitate the behaviour of any real system or phenomenon (either physical or 
social) at the time that provides with an adequate framework to perform highly effective learning experiences. 
The main goal of an ES is to promote the learning by discovery and to enhance the skills involved in the study of 
a certain system/phenomenon of interest. ES provides to students with the opportunity to interact, reflect and 
learn, so, they actively participate in the educational process (Aldrich, 2005; Valverde, 2010). 

According to Aldrich (2005), the success of any educational experience through simulations depends on the 
proper configuration of the following three elements: the Simulation component, the Pedagogical component 
and the Game component. See Fig. 1. 
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Figure 1. Educational Simulation components 


With regards to the Simulation component, it allows recreating and representing the real world in different 
circumstances and with different levels of abstraction and realism. More precisely, the term Simulation in this 
paper refers to the "process of designing a model of a real system and carry out experiences with it, with the 
purpose of understand the system behaviour and evaluate new strategies -within the limits imposed by a 
criterion or a set of them- for the operation of the system." (Shannon & Johannes, 1976). Some of the main 
features of any type of simulation are: 

• Useful for making decisions. 

• Enables to evaluate and/or predict potential future solutions in today's present. 

• Enables to optimize the system behaviour (if the system can be modified, e.g., a factory plant). 

• Offers a safe and economical method for training operators. 

• Enables to explore situations where actual experimentation might be dangerous, problematic, 
expensive or directly impossible. 

Therefore, the Simulation component in a learning experience allows taking people to a time or place that they 
are unable or unlikely to experience directly, thus encouraging the learning within artificial situations that can 
be fully adapted to enhance the learning experience. In addition, the above features can be usually 
complemented with the capability of the simulators to represent and connect huge amounts of information 
through multimedia, such as high-definition video and high-fidelity surrounding sound (Alessi & Trollip, 1991; 
Gibbons & Fairweather, 1998), which altogether add an extra value to the ES and makes the learning experience 
more attractive to the students. 

Regarding the Pedagogical component, one important feature of an ES experience is the opportunity to apply 
the Learning by Doing (LbD) concept in classroom. Note that according to Martinez (2003), "It is a 
misconception that people learn by listening or reading and validate your knowledge through an examination 
(where his memory is measured but never his understanding). People learn by doing, making mistakes and 
reflecting on how to solve problems, usually with the help of someone more experienced". 

Within the European Higher Education Area (EHEA) paradigm the university students not only learn knowledge 
in classroom, they must also acquire the general competencies that they will need when transferred to the 
labour market after completing their studies (EHEA, 1999). In this context, the term competency means, as 
defined by Echevarria (2001), the ability to discriminate the information that is important to cope with a certain 
situation/problem and the ability to use this information to face with it. Note that the concept of competency is 
sometimes referred as skill in the Anglo-Saxon context (Bonson, 2009); in this paper the word skill is used with 
the meaning of methodological capability to apply the technical knowledge to specific real situations. In this 
sense, who possesses professional competency is who has the necessary knowledge, skills and attitudes to 
exercise their own work, to solve problems creatively and with independency, and that is able to contribute to 
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their work environment and work organization (Lasnier, 2000; Biggs, 2005; Tardiff, 2006). Therefore, since the 
development and assessment of competencies should be performed in realistic environments (Lussier & Allaire, 
2004; Airasian, 2001), the ES can contribute (through a LbD approach) to the design of new activities that guide 
to the students towards the improvement of their professional competencies as required for their expected 
future positions. 

Finally, the Game component of an ES experience motivates the student during the learning process through 
amusing activities that indirectly provide benefits. Note that in a game situation the player engages in a 
lose/win situation that requires the practice of knowledge, skills and techniques (i.e., competencies) that are 
either already acquired or in the process of being acquired. Practice is said to facilitate knowledge acquisition 
(Biggs & Moore, 1993). 

In this paper an ES case study is introduced in which a teaching experience has been performed based on the 
use of a flight simulator. The targets of this experience have been the students of Bachelor in Aeronautical 
Management, within the subject of Technologies and Procedures for Air Traffic Control and Navigation (note, 
however, that the same or similar exercises can be adapted to other fields of study, e.g.. Aeronautical 
Engineering). One of the difficulties of this subject is that it introduces several new Air Traffic Management 
(ATM) concepts that are currently under development in the context of the Single European Sky (SESAR 
Consortium, 2007) and that are difficult for the students to understand in deep only from the point of view of 
theoretical explanations. For instance, the SESAR ATM paradigm requires the use of 4D trajectories, i.e., precise 
execution of a flight trajectory in the three dimensions and time. The students should note during the course 
that the precise 4D navigation is not affordable in practice for humans (i.e., it requires high levels of automation 
in all the control and navigation processes), and the best way to allow them observing that fact is through a 
practical flight activity (i.e., learning by doing) in which they adopt the role of a pilot. After the experience, the 
students have been questioned with a knowledge evaluation exam and a satisfaction survey requesting their 
opinion. 

The paper is structured as follows. Section 2 outlines the design process of the teaching experience and 
activities proposed to the students taking into account the competencies that the student should acquire and 
the relationship with the contents of the subject. In addition, the last subsection of the section 2 shows how the 
teaching experience has been executed and assessed, whereas Section 3 explains the results obtained after the 
assessment. Finally, Section 4 reads the conclusions. 


2 PLANNING AND DESIGN OF A TEACHING EXPERIENCE THAT USE A FLIGHT SIMULATOR 

The planning and design of the teaching experience have been divided in the definition of objectives, 
methodology applied and assessment methods. 

2.1 Definition of the objectives 

The main goal of the teaching experience proposed here is to guide the students in the process of acquiring a 
set of specific and transversal competencies that have been previously defined in the context of the subject 
"Technologies and Procedures for Air Traffic Control and Navigation", a subject in which the students are 
introduced to the current and future technologies and procedures used in aviation. Some of the contents of this 
subject are difficult to be understood in deep only through theoretical explanations, e.g., how the students can 
understand the difficulties of performing a precise 4D navigation if they have not experienced the difficulties of 
the -much more easier- 2D and 3D navigation? Therefore, the purpose of the learning activities proposed in 
this paper is to allow the students to play the role of a pilot by means of a flight simulator, so they can 
understand the basic principles of the air navigation, experience the most common tasks performed by pilots 
and to observe how the new technologies can facilitate and transform such processes. 

Some of the pursued competencies that the students should acquire in the context of the subject (specifically 
related to the Navigation sub-topic of the subject) have been identified in Table 1, together with the 
identification of the specific objectives related to each of the competencies that the students should be able to 
perform after the teaching experience. The description of these specific performance objectives can be found in 
Table 2. 
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Competency 

identifier 

Competency 

Typology 

Competency 

components 

Objective 

s 

Cl 

Deep understanding of all the processes 
involved in the air transportation system 

Specific 

Knowledge 

01 

Skills 


Attitudes 

02 

C2 

Deep understanding of the reasoning, 
abilities and worries of flight pilots 

Specific 

Knowledge 

03 

Skills 

04 

Attitudes 

05 

C3 

Capability to detect improvement areas 
at any process of the air transportation 
system 

Specific 

Knowledge 

06 

Skills 


Attitudes 

07 

C4 

Capability to identify new potential 
technologies and procedures that may 
improve the command and/or 
navigation of a flight 

Specific 

Knowledge 

08 

Skills 

09 

Attitudes 

010 

C5 

Capability to have an open, tolerant and 
adaptive thinking 

Transversal 

Knowledge 


Skills 

Oil 

Attitudes 

012 

C6 

Capability to make good decisions in 
different situations 

Transversal 

Knowledge 

013 

Skills 

014 

Attitudes 

015 


Table 1. Competencies and relationship with the teaching experience objectives 


Objective 

identifier 

Specific performance objectives 

01 

The student knows the concepts and complexities related to the control and navigation of 
an aircraft during a flight 

02 

The student has enhanced interest for the contents of the subject 

03 

The student knows the different types of instrumentation on-board to command an 
aircraft 

04 

The student have basic abilities for the commanding and navigation of an aircraft 

05 

The student is able to have enhanced empathy towards the pilots and knows the feelings 
related to flying 

06 

The student understand the tasks required for the commanding and navigation of an 
aircraft as a particular process within the complex air transportation system and thus is 
able of evaluating the impact of any technological or procedural change that may 
improve or worsen the command and navigation of an aircraft 

07 

The student is able to perform critical thinking with regards aeronautical technologies and 
procedures 

08 

The student knows the historical evolution of the technologies and procedures used in the 
commanding and navigation of aircraft and is able to evaluate their impact in the flight 
processes 

09 

The student can manipulate the different technologies and instrumentation on-board and 
can follow basic flight procedures 

010 

The student is able to perform critical thinking with regards aeronautical technologies and 
procedures 
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Objective 

identifier 

Specific performance objectives 

Oil 

The student is able of adapting rapidly to a new environment (in this case, a cockpit) while 
processes a lot of information and puts in practice his/her psychomotor coordination skills 

012 

The student is able to have enhanced empathy towards the actors of any working position 
(in this case, the pilot's working position) 

013 

The student has good knowledge about the context in which the decision-making is 
performed (in this case, the knowledge of Cl, C2, C3 and C4) 

014 

The student is able to identify how a decision may affect to, or may be affected by, 
different aspects of the same problem (in this case, how a decision may affect to or may 
be affected by the processes conducted by the pilot or the navigation systems of an 
aircraft) 

015 

The student is able to feel self-confident during the decision making process and also is 
able to have enhanced empathy towards the main affected actors (in this case, the pilots) 


Table 2. Performance objectives of the teaching experience 


2.2 Methodology 

To reach the pursued goals, a set of activities have been designed for the students, following the approach 
proposed by (Boix & Armisen, 2008) and summarized in Figure 2, in which the main characters, teachers and 
students, must interact with each other so that the second can learn and acquire the due competencies 
necessary to become a good professional. 


The student must acquire 


Making significant 


Activities that are in 

competencies 


learning activities 


relation with contents 


Figure 2. Methodology for the design of learning activities oriented to competencies 


These learning activities have included the usage of an ES platform (i.e., a flight simulator) so the students can 
acquire the pursued competencies by means of a LbD approach. Therefore, the three ES components (i.e., 
Pedagogical, Game and Simulation) have been set in the following three sequential steps for the effective 
performance of the teaching experience. 

2.2.1 Pedagogical component 

First, the relationship between the contents of the subject and the specific performance objectives and 
competencies has been established (i.e., Pedagogical component of ES). Table 3 identifies some of the contents 
of the subject of Technologies and Procedures for Air Traffic Control and Navigation and puts them in relation 
with the competencies and objectives pursued. These objectives have been sorted in the table attending to the 
component (i.e., knowledge, skills or attitudes) of the related competencies. 
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Subject 

identifier 

Subject content 

Competency 

components 

Competencies 

related 

Objectives 

related 

SCI 

Concepts, procedures and 
technologies applied to the 
air navigation 

Knowledge 

Cl, C2, C3, C4, C6 

01, 03, 06, 08, 013 

Skills 

C6 

014 

Attitudes 

Cl, C3, C4, C6 

02, 06, 09, 014 

SC2 

Instrumentation available 
on-board for the navigation 
and control of an aircraft 

Knowledge 

Cl, C2, C3, C4, C6 

01, 03, 06, 08, 013 

Skills 

C2, C4, C6 

04, 09, 014 

Attitudes 

Cl, C3, C4, C6 

02, 06, 010, 015 

SC3 

Practical use of cockpit 
instruments and navigation 
procedures 

Knowledge 

Cl, C3, C6 

01, 06, 013 

Skills 

C2, C4, C5, C6 

04, 09, Oil, 014 

Attitudes 

Cl, C2, C3, C4, C5, 

C6 

02, 05, 07, 010, 
012, 015 

SC4 

Practical understanding of 
the multiple tasks and 
difficulties arisen at the 
pilot's working position 

Knowledge 

Cl, C2, C3, C6 

01, 03, 06, 013 

Skills 

C5, C6 

Oil, 014 

Attitudes 

Cl, C2, C3, C4, C5, 

C6 

02, 05, 07, 010, 
012, 015 

SC5 

Deeper and practical 
understanding about the 
difficulties of executing 4D 
trajectories (SESAR context) 

Knowledge 

Cl, C2, C3, C4, C6 

01, 03, 06, 08, 013 

Skills 

C2, C4, C5, C6 

04, 09, Oil, 014 

Attitudes 

Cl, C2, C3, C4, C5, 

C6 

02, 05, 07, 010, 
012, 015 


Table 3. Subject Contents and their relationship with the competencies and objectives 


2.2.2 Game component 

After the definition of the Pedagogical component, the activities to be executed by the students in classroom 
have been developed (i.e.. Game component of ES). The idea is to design some exercises that make use of a 
flight simulator so that the students can adopt the role of a pilot and thus assimilate in practice the theoretical 
concepts seen in class and improve their skills while at the same time they can enjoy of the learning process, 
thus generating positive attitudes towards the subject and the concepts learnt. Attending to the Pedagogical 
component stated in Table 3, i.e., competencies to be acquired and their relation with the subject's contents, 
the following activities have been designed: 

• Piloting experience. Initially, during the first contact with the flight simulator, the activities must be 
driven to allow the students to get used to the controls and manoeuvrability of the aircraft when it is 
airborne. The first activity requires the student flying straight while maintaining constant altitude and 
also maintaining constant other certain flight parameters. Later, when the students can stabilize the 
aircraft, they can start to manoeuvre the aircraft, performing some controlled turns and changing the 
flight level by ascending and descending the aircraft with a controlled vertical speed. These exercises 
are not trivial for beginners and they present different levels of difficulty depending on the aircraft 
type used (in general, the bigger the aircraft the more difficult its stabilization during the flight). 

• Air navigation, from theory to practice. This activity aims at showing the historical evolution of the air 
navigation, a problem that involves two unknowns, i.e., which is the current position of the aircraft and 
how to reach a desired destination in the most optimal way possible. In the first place, the students are 
requested to use charts and visual references to know where they are while still commanding the 
airborne aircraft (as pilots traditionally did in the past and still sometimes do). Later they are 
introduced to the radio-navigation instruments available on-board: ADF (Automatic Direction Finder), 
which makes use of the ground navigation aid called NDB (Non Directional Beacon), and VOR/DME 
(Very High Frequency Omnidirectional Range/Distance Measure Equipment); see Martinez and Belda 
(2000). Such exercise allows the students to have a more practical view of the navigation concepts 
seen in the theoretical classes, and also having a better "in situ" understanding of how these 
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technologies actually help the pilots in their tasks, specially during severe climatologic conditions (e.g., 
dense fog or severe thunderstorm, among others), which are also introduced in the exercises. 

• Modern technologies for flight control and navigation. Once the students have understood (and lived) 
the difficulties related to the control and navigation of an aircraft, they are introduced to the most 
modern technologies developed to aid pilots in those complex tasks. In particular, they are able to 
manage a navigational system that use Global Navigation Satellite Systems technologies (i.e., GNSS, 
which include GPS, Galileo and GLONASS satellite constellations), that combined with the radio¬ 
navigation and inertial systems altogether provide to the pilots with higher accuracy of the flight 
tracking information, with more interdependency with respect to the ground radio-navigation aids and 
with an improved displaying of the information and pilot-system interaction. Finally, the students are 
able to activate and configure different types of autopilots, thus observing how easy it turns the 
execution of the same tasks that they previously did manually with difficulties in the above activities 
(note that some of the current autopilots are able to land an aircraft with very little/almost null 
intervention of the pilot). After that, the students are more sensitive to understand the complexity and 
value of the incoming 4D navigation systems, which are currently being under development 
introducing even higher levels of automation to enable a future air traffic management system (i.e., 
SESAR) which aims at being more precise, safer and efficient than the current one. 

Table 4 shows the relationship among the activities and each of the contents and objectives related. Note that 
the three activities designed are related directly or indirectly with all the subject contents and thus will all the 
performance objectives. It occurs because all these contents are unavoidably present during the three flight 
simulation activities. However, the sequence order of these activities allows introducing more subject contents 
and extra milestones difficulty in a progressive way, and until all the subject contents are fully -and 
consciously- managed by the students. 


Activity 

Activity 

Contents directly 

Contents 

Objectives 

identifier 

related 

indirectly related 

related 

A1 

Pilodng experience 

SC2, SC3, SC4 

SCI, SC5 

All 

(directly or indirectly) 

A2 

Air navigation, from theory to 
practice 

SCI, SC2, SC3, SC4 

SC5 

All 

(directly or indirectly) 

A3 

Modern technologies for flight 

SCI, SC2, SC3, SC4, 


All 

control and navigation 

SC5 


(directly) 


Table 4. Activities and their relationship with the subject contents and objectives 


2.2.3 Simulator component 

As a third step, the components to install a functional flight simulator have been chosen and configured (i.e., 
Simulator component of ES). Note that the diversity of software and hardware components available nowadays 
to perform a flight simulation for training purposes is huge (Aguado & Ruiz, 2013) and that many different 
components have been designed to set up different kind of Flight Simulation Training Devices (FSTDs) in order 
to provide to the users with different functionalities, qualities and with different levels of realism (EASA, 2008). 
Thus, the selection of the most appropriate combination of software and hardware for the purposes of the 
teaching experience must take into account the actual needs to perform the activities in the classroom. 

For instance, some of the FSTDs are used to train pilots (civilian and military) in the commanding of different 
kinds of aircraft under different circumstances, thus requiring software and hardware able to represent the 
aircraft dynamics and the boundary conditions of real operations with maximum realism (Sanchez, 2012). In 
such cases, it might be recommendable to use a Full Flight Simulator (FSS), which is a perfect replica of a certain 
aircraft cockpit that includes full realistic motion and maximum quality and realism in the sound and video 
recreated. Of course the price and the complexity of both the software and the hardware is very high as well as 
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the difficulty to operate the flights, which altogether suppose a level of realism that must be considered 
"excessive" for the purposes of this teaching experience. 

Nonetheless, since the target of the experience is inducing the students to feel like walking in the pilot's shoes, 
the appropriate flight simulator must still include an acceptable level of realism for the representation of the 
flight dynamics under different weather conditions and also include a minimum cockpit representation to allow 
the practice of the most common procedural aspects (and for which a maximum level of realism in the 
simulations is not required). This kind of flight simulator is referred in EASA (2008) as a Basic Instrument 
Training Device (BITD), which are much more cheaper than FSSs, easier to set up and also easier to command, 
thus being ideal for the purposes of this teaching experience. 

Furthermore, there are others requirements that are also important to design and set up the flight simulator 
platform. In particular, the system must be useful for different purposes, including (but not limited to) the 
teaching experience proposed that should allow the users reaching the learning goals through the 
experimentation of the role of a pilot, it must be efficient, meaning that it should allow the users reaching the 
goals defined and the role experimentation of a pilot with the minimum (economic) resources, it must be easy 
to use and ergonomic in order to allow a fast learning curve for its usage, and it must have a reasonable cost 
(including installation) with regards to the academic objectives pursued. Finally, the system must be reliable, 
i.e., the probability of failure or error must be the lowest possible; robust, so it can stand the test of the time 
and the usage by different users (in particular, students); and polyvalent, in order to can be used for different 
kind of teaching and research activities. 

After a comprehensive analysis of the market options, and taken into account all the above requirements, the 
software chosen to perform the flight simulations has been the Microsoft Flight Simulator X, whereas the 
hardware components have been a medium-power computer (Intel i5 CPU with 4GB of RAM and with a 
dedicated graphical card featuring FIDMI out), a 24 inches high-resolution flat screen, and the specific flight 
simulator component "Saitek Pro Flight System", which is composed by a yoke and throttles that are robust, 
ergonomic, easy-to-install and present a good quality-price relationship. Rudder pedals have not been found to 
be necessary for the purpose of this teaching experience, although their introduction could be considered to 
increase the realism during the practice of the flight exercises. See Figure 3. 



Figure 3. Flight simulator cockpit view (Yoke and throttle). Flight instruments are software-simulated thus 
enabling different cockpits and views, e.g.: Cessna 172 (left), Boeing 737 (right) 


2.3 Assessment methods for the teaching experience 

The evaluation of competencies requires the alignment and combination of different types of assessment 
methods, due to the different nature of the components that compose the competencies, i.e., knowledge, skills 
and attitudes (Yaniz & Villardon, 2006; McDonald, Boud, Francis & Ginczi, 2000). This approach is often referred 
as Authentic Assessment (De Miguel, 2005; Herrington & Herrington, 1998). In order to evaluate this teaching 
experience and the degree of achievement of the goals pursued (Tables 1 and 2), four different assessment 
methods have been considered: an objective test of knowledge, an execution test, observation techniques, and 
an attitude and opinion survey. Each assessment method is further explained in the following subsections, and 
Table 5 shows which of the objectives have been evaluated with each of the assessment methods. 
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Assessment method 

Ol 

02 

03 

04 

05 

06 

07 

08 

09 

010 

Oil 

012 

013 

014 

015 

Objective test 

X 


X 



X 


X 





X 

X 


Execution test 

X 



X 


X 



X 


X 


X 



Observation 

techniques 


X 



X 






X 

X 



X 

Attitude and opinion 
survey 


X 





X 



X 




X 

X 


Table 5. Assessment methods and objectives assessed 


2.3.1 Objective test 

A short objective test has been prepared and filled by the students in order to evaluate whether they have 
assimilated and learnt the concepts explained during the flight simulation activity, i.e., to evaluate the 
knowledge component of competencies Cl, C2, C3, C4 and C6 (also the skills component of C6 is evaluated). 
The test is a multiple-choice test with five questions and four possible answers each (only one is correct). The 
test can be seen found in the following table 6. 


Question 

Statement 

1 

The flight basic instruments which provide altitude indication and heading to the pilot, are 
respectively: 

a. Altimeter and Attitude indicator (Artificial horizon). 

b. Altimeter and Vertical speed indicator (Variometer). 

c. Altimeter and Heading indicator (Directional gyro). 

d. Airspeed indicator and Heading indicator (Directional gyro). 

2 

What does Air Navigation mean? 

a. The science and technology that aims to determine the height of an airplane 
relative to the ground and maintain exactly the desired route. 

b. The science and technology that aims to determine the position of an aircraft 
relative to the surface of the earth and accurately maintaining the desired path. 

c. The science and technology that aims to determine the position of an airplane. 

d. The science and technology that aims to keep exactly the desired route of an 
airplane relative to the surface of the earth. 

3 

Two of the some radio navigation aids that provide information to the pilots for carrying 
out a safe flight under adverse meteorological conditions (fog, thunderstorms, blizzard, 
etc.), are: 

a. Airspeed indicator and VOR/DME. 

b. ADF and VOR/DME. 

c. ADF and HSI. 

d. VOR/DME and Variometer. 

4 

What system will allow to airplanes to being totally independent from the ground 
stations? 

a. GPS navigation system. 

b. Galileo navigation system. 

c. GLONASS navigation system. 

d. All the above. 

5 

A satellite-based navigation system will allows... 

a. Airspace "optimization". 

b. Cost reduction. 

c. Reduced flight time. 

d. All the above. 


Table 6. Knowledge test 
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In order to asses the effectiveness of the teaching experience to enhance the knowledge of the students the 
test has been requested to the students six weeks after the flight simulation experience without previous 
notification (the test has been used to assess the teaching experience only, thus it did not count to pass the 
course). 

2.3.2 Execution test 

The execution test has been prepared with a set of milestones to be accomplished by the students during the 
flight simulation as a way to assess the skill component of the competencies C2, C4 and C5, and also the 
knowledge component of the competencies Cl, C3 and C6. The flight instructor (i.e., the teacher) has been 
guiding and assessing the correct achievement of the milestones during the execution of the activities seen in 
Table 4. In addition, the flight instructor has been giving the necessary information at each precise moment of 
the activity, in order to facilitate to the students interiorizing the knowledge, skills and techniques, and attitudes 
pursued. See Fig. 4. 

The flight simulation activities have been performed with two students per machine, one of the students of 
each group has been commanding and navigating the aircraft for a while, whereas the other has been observing 
and helping as a co-pilot (they have exchanged positions during the activity). 



Figure 4. The teacher has guided and assessed the correct execution of the activities 

2.3.3 Observation techniques 

The introduction of observation techniques is an interesting assessment method considering that the 
evaluation of some of the competency components pursued in this teaching experience, in particular the 
attitudes, are difficult to evaluate with objective or execution tests, such as the level of interest of the student in 
relation with the contents of the subject, the understanding of the feelings of a pilot during a flight or the level 
of self-confidence of the student towards the pilot's working position (i.e., competencies Cl, C2, C5 and C6). 
Therefore, these competency components have been partially evaluated through some observation techniques 
applied during the execution of the flight simulation exercises (the skill component of C5, i.e., ability to rapidly 
adapt to a new situation has been also evaluated through observation). In particular, the teacher has been 
aware about the comments spontaneously verbalized by the students in order to find traces that reveal the 
presence and/or development of the pursued attitudes. When the students have not spontaneously made such 
comments (or reactions) the teacher has requested the feedback about the current feelings, opinion and 
expectative in order to perform the assessment. 
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2.3.4 Attitude and opinion survey 

Finally, the same 38 students that did the exercises of the teaching experience have been also requested to fill a 
short survey in order to assess some of their attitudes (in particular the attitude components of Cl, C3, C4 and 
C6) and also their opinions towards the flight simulator activity and their elements. The complete survey with 
16 questions can be found in the following table 7. The rating scale for the survey has been defined as follows: 1 
= Strongly disagree or very low rating; 2 = Disagree or low rating; 3 = Neutral; 4 = Agree or high rating; 5 = 
Strongly agree or very high rating. 


Items/questions to evaluate 

1. The Flight Simulation (FS) activity has helped to me to understand the subject's concepts better. 

2. The FS activity has increased my interest for the subject. 

3. The theoretical content of the activity is well structured. 

4. The theoretical part is suitably integrated with the practice part. 

5. FS practice has helped to me to learn new things. 

6. 1 have found interesting the FS activity. 

7. The FS software has been useful for doing the activity. 

8. The FS hardware (yoke and throttles) has been useful for doing the activity. 

9. Value the degree of difficulty of the exercises performed. 

10. Value your level of satisfaction with the activity. 

11. Extra time is necessary to develop the activity better. 

12. After doing the FS activity 1 have increased my general knowledge regarding the subject. 

13. The teacher/instructor of the FS activity has expressed with clarity. 

14. The teacher has used useful examples to explain the subject. 

15. The teacher has shown enthusiasm for the subject. 

16. Do you think that is necessary more content as seen in this activity in the Aeronautical 
Management Degree? 


Table 7. Attitude and opinion survey 


3 RESULTS 

The following subsections provide the results obtained for each of the assessment methods. 

3.1 Objective test 

The objective test has been answered by 38 of a total of 40 students that performed the Flight Simulation 
activity. The results obtained show that the 92,11% of the students (i.e., 35 students) have successfully passed 
the test, while only 3 students have failed the test with a mark of 2 out of 5 (three questions answered correctly 
have been considered the threshold to pass the exam). The most frequent marks have been "3" and the "4" 
with a number of students of 13 and 20 respectively. Only three of them have got the maximum punctuation, a 
mark of 5. 

The results can be analysed question by question. The first query has been answered correctly by 28 of the 38 
students, which means a success rate of 73,68%. The number of students that have answered correctly the next 
second, third and fourth questions have been 22, 15 and 34 respectively, and its corresponding success rates 
are 57,89%, 39,47% and 89,47%. The fifth and last question has been answered correctly by 37 of 38 students, 
which means a success rate of 97,37%. 

Note that it was expected from the beginning that the second and third questions were the most difficult to 
answer, since these queries show ambiguous answers due to the use of acronyms in the third and due to the 
use of tricky definitions in the second. Nevertheless, the second question was expected to be the most failed 
question and not the third one as finally occurred. 

It is important to point out that this test has been requested to the students six weeks after the flight simulation 
experience and they were not informed that the test was going to be requested (the test has been used to 
assess the teaching experience only, thus it did not count to pass the course). 
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The same test has been requested to other two control groups in order to contrast the test results with other 
students that have not carried out the learning experience, thus assessing the effectiveness of the teaching 
experience. The first of these control groups has been formed with students that are not related to Aeronautical 
Management studies. This group has been referred as Group X: Non-AM. The second control group has been 
composed with students of Aeronautical Management that have still not taken the course of Technologies and 
Procedures for Air Traffic Control and Navigation and thus they have not learnt from the teaching experience 
presented here. This group has been referred as Group Y: AM pre-experience. Finally, the group that has been 
learning with the teaching experience and that after has been evaluated is referred as Group Z: AM post¬ 
experience. Table 8 shows for these three groups the statistical summary of the test results. 


Statistical parameters 

Group X: 

Non-AM students 

Group Y: 

AM Pre-experience 

Group Z: 

AM Post-experience 

Sample size, n x 

n x = 39 

n y = 41 

n z = 38 

Sample mean, X 

X = 1.8462 

y = 2.4878 

Z = 3.5789 

Sample variance, S x 

S 2 X = 1.0810 

S 2 y = 1.2061 

S 2 z =0.5747 

Proportion of passing 
marks, p x 

p x = 28.21% 

p y =53.66% 

P z = 92.11% 


Table 8. Statistical summary of the three group samples 


Hypothesis 
test id 

Ho 

Hi 

Contrast statistic 

p-value 

Result 
(a =0.03) 

HT1 

Py - Px — 0 

Py > P X 

Z=—- Py P * --2.3116 

'' n +n 

p.u-p.h; „) 

' y x 

x n x . Px+n .p 

P °= n +n 

x y 

0.0104 

H! • Py> P x 

HT2 

Pz-Py<0 

Pz>Py 

p —p 

7 — z v — 3 8DQD 

0.0001 

H^P^Py 

p " )( 

„ n z .p z +n.p 
p 0 = 

n +n 

z y 

HT3 

Pz - Px — 0 

Pz>Px 

Z=—-'---5.7163 

x n z .p z +n x .p x 

Po= n + n 

Z X 

0.0000 

H 1 :Pz>Px 

HT4 

Hy-U,< 0 

Py > Px 

V-X 

-= 2.6844 

1 5 2 5 2 

1 - + - 
n n 

1 y x 

0.0036 

Hi: py > p x 

HT5 

ll,-Uy< 0 

Pz > Py 

Z- Z ~y -5.1702 

S 2 S 2 

V —+— 

1 n z n y 

0.0000 

Hi: p z > py 
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Hypothesis 
test id 

Ho 

H\ 

Contrast statistic 

p-value 

Result 
la =0.03) 

HT6 

0 

Pz > Px 

Z= , Z ~ X =8.3718 

! s 2 S 2 

Z . X 

1 n z n x 

0.0000 

Hi : pz > Px 

HT7 

Oy=0 X 

Oy^O X 

s 2 

F- — =1.1158 

s x 

0.3682 

H 0 : 0y=0x 

HT8 

a z <a y 

O z >Oy 

s 2 

F- — = 2.0987 

s? 

0.0124 

Hi ; 0 z >0y 

HT9 

o z <o x 

Oz>0x 

S 2 

F-— =1.8810 

0.0286 

Hi : 0z>0 x 


Table 9. Hypothesis tests comparing the results of the three groups 


According to Table 8 important differences can be observed between the performances of the three groups. 
Table 9 shows the statistical results for different hypothesis tests (Newbold, 2001) that make comparisons 
among the three groups with regards to the proportion of students that have passed the exam (i.e., HT1, HT2 
and HT3), the average number of correct answers (i.e., HT4, HT5 and HT6), and the variance that describes the 
variability of correct answers within each group (i.e., HT7, HT8 and HT8). 

Thep-values that have been calculated indicate that all the null hypotheses, H 0 , can be rejected with a 
significance level of a = 0.03, thus having strong statistical confidence in the rejection of the hypothesis (97% of 
confidence), except in the case of HT7 that should be accepted even for much higher significance levels (up to a 
= 0.6882). The null hypothesis, H 0 , can be rejected in favour of the alternative hypothesis Hi for any value of the 
contrast statistic higher than the significance level a and with a confidence level of y = (1 - a )-100%. The p- 
value of the contrast can be interpreted as the critical significance level value such that if a> p-value the null 
hypothesis H 0 can be rejected (with a confidence level of y= (1 - a )-100%) and if a< p-value cannot (in such a 
case it is often said that H 0 is "accepted"). In the cases shown in Table 9 all the null hypotheses can be rejected 
with a significance level a = 0.03 and with a confidence level of 97% (they some of them can be rejected even 
for higher confidence levels), except in the case HT7 in which H 0 cannot be rejected with such level of 
confidence (thus it should be accepted). 

These results mean that there is strong statistical evidence that confirm the differences among the three 
groups. Only a 28.21% of the students of Group X have successfully passed the exam, whereas a 53.66% of the 
students of Group Y have done so. This result suggests that all the students of Aeronautical Management 
already had previous knowledge about some of the concepts questioned in the test. Nevertheless, the fact that 
the 92.11% of the students that have taken the teaching experience have passed the test, together with the fact 
that the average number of correct answers, z, is significantly higher than x and y, suggests that the teaching 
experience has positively enhanced the knowledge of the Group Z students. In addition, another finding that 
should be remarked is that the variability of the groups X and Y, i.e., S 2 x and S 2 y , are statistically equal, while the 
variability of Group Z, S 2 , is significantly smaller. This fact suggests that after the teaching experience the 
knowledge of the students has converged (i.e., the knowledge seems to have been unified among all the 
students) while the number of questions correctly answered "by-chance" seems to have been reduced. Figure 5 
summarizes these results in a graphical manner. 

The high success rate of the students passing the exam indicates that objectives 01, 03, 06, 08, 013 and 014 
have been successfully achieved and as a consequence the competencies related have been enhanced. 
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GroupZ 

(AM Post-experience) 


GroupY 

(AM Pre-experience) 


Group X 
(Non-AM) 


Figure 5. Box diagram with the statistical results of the three compared groups 

3.2 Execution test 

During the teaching experience all the students have been able to follow all the exercises proposed, although 
with high heterogeneity with regards to the levels of precision during the execution of the manoeuvres (i.e., 
some of the students were already used to manage flight simulators whereas for other it was their very first 
contact). Therefore, the successful execution of the exercises designed indicates the consecution of the 
objectives 01, 04, 06, 09, 011 and 013. 

3.3 Observation techniques 

In general, all the students have shown high levels of concentration during the activities and explanations 
whereas they seemed to enjoy the experience as expected (see Fig. 6). This fact, together with the verbal 
feedback obtained, suggests that the objective 02 has been achieved (i.e., the students showed interest for the 
subject) and thus the competency related, i.e., Cl, has been successfully enhanced for most of the students. 
During the execution of the activities, many students spontaneously verbalized some attitude changes 
regarding the pilot working position, with sentences such as "I did not know that the process of flying required 
to be so calmed down, whereas the speed of the flight is so fast". The identification of this kind of sentences 
during the execution allows stating that the student can now understand in a deeper way the tasks, feelings and 
logical thinking of the pilots (objectives 05, 012 and 015). When the students did not spontaneously express 
their feelings, the teacher directly requested their opinion and thinking. All the students, except the ones with 
previous experience in flight piloting, admitted a deeper understanding of the feelings, reasoning and worries 
related to the pilot's working position after the simulation experience. 

At the beginning of the exercises some of the students were notably anxious by the fact of being in the working 
position of a pilot by their very first time, a feeling that was emphasised by the fact that they had to process a 
lot of new information, had to learn very fast how to manipulate the command and navigation instruments and 
at the same time they had to coordinate all the actions required to accomplish the milestones of the activities. 
After some minutes of (guided) practice, it was observed that all the students were able to adapt to the new 
situation while most of them showed higher levels of self-confidence than at the beginning of the activities. 
These observations suggest that objectives Oil and 015 have also been achieved successfully. 



0 1 2 —3““ 4 T - 6 
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Figure 6. The students presented high levels of concentration 


3.4 Attitude and opinion survey 

To facilitate the assessment of the attitudes and opinions generated by the teaching experience, the questions 
of the survey can be re-grouped an analysed in groups of related meanings. 

Questions 7, 8 and 14 are fairly illustrative of the satisfaction levels regarding the Simulation component of the 
teaching experience (see Fig. 7). In particular, questions 7 and 8 show that most of the students found the flight 
simulation system (both hardware and software) to be highly appropriate for the purposes of the activities. 
Specifically, the 97% of the students agreed or strongly agreed with the question 7 and the 89% with question 
8, whereas the rest remained neutral. Question 14 shows that the major part of the students has considered 
useful the ability of the flight simulation system to recreate realistic scenarios that were used for educational 
purposes. Specifically, the 82% of the students agreed or strongly agreed, whereas only a 5% disagreed and the 
rest remained neutral. 

Note that to answer these questions (i.e., 7, 8 and 14) the students have needed to analyse which of the 
aeronautical technologies available in the cockpit have been used during the exercises, and then determine the 
levels of realism provided by the simulation components in relation to the activities proposed during the 
teaching experience. The high levels of satisfaction obtained in these items suggest that the students must have 
taken into account the usefulness of abstracting some of the instruments and procedures required in real flight 
executions (the simulation components are part of a Basic Instrument Training Device and thus they are 
relatively simplistic and austere) in order to ease a basic training of the commanding and navigation skills. Such 
(most likely) reasoning indicates that the students are now able to perform critical thinking with regards the 
aeronautical technologies present in the cockpit, and thus the objectives 07 and 010 (i.e., critical thinking) can 
be considered as achieved. Under the same arguments, also the objective 14 (i.e., ability to identify how a 
decision may affect or may be affected by the pilot or the cockpit instrumentation) should be considered as 
successfully observed. 
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Question 7: 

Question 8: 

Question 14: 

'The Flight Simulator software 

“The Flight Simulator 

"The teacher has used useful 
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Figure 7. Survey questions related to the Simulation component 


Questions 2, 6 and 9 illustrate the evaluation with respect the Game component of the teaching experience 
(see Fig. 8). In particular, question 6 shows that most of the students have found the activity to be interesting 
(90% agreed or strongly agreed), while question 2 shows that students have increased their interest on the 
subject due to such activity (82% agreed or strongly agreed). These results are directly related with the 
successful consecution of objective 02 and confirm the assessment done through observation of attitudes 
during the execution of the activities. Question 9 shows that the difficulty of the flight simulation exercises has 
been considered adequate in average: the 53% considered neutral/medium difficulty, the 24% considered the 
exercises to be difficult (although no one found them very difficult), and the 23% considered the exercises to be 
easy or very easy (note that some of the students were familiar to flight simulators whereas for some of them 
this activity was their very first contact). Note that the perception of the difficulty is a direct reflex of the self- 
confidence levels regarding the flight navigation activities performed. The fact that the levels of difficulty 
perceived are well balanced in average suggests that the competences related to objective 015 (i.e., ability to 
feel self-confident in the context of flight commanding and navigation) have been achieved. 


Question 2: 

“The Flight Simulation activity 
has increased my interest for 
the subject" 



Question 6: 

"I have found interesting the 
Flight Simulation activity" 

0 * 

1% 


Question 9: 

"Value the degree of difficulty 
of the exercises performed" 


OK 



Figure 8. Survey questions related to the Game component 


Regarding the Pedagogical component, the questions 1, 5 and 12 allows stating that the students have 
perceived the activity as valuable from the point of view of the learning process (see Fig. 9). Note that in 
questions 1 and 5 more than 75% of the students recognized that the activity has contributed or strongly 
contributed to their learning process, whereas in question 12 (that refers to the amount of knowledge acquired 
in relation to all the contents of the subject) this percentage has diminished up to 63%. This could be attributed 
to the fact that the flight simulation activity represents no more than 5% of the concepts seen in the subject 
during the entire course. In any case, these results contribute to reinforce that the objective 01, 02, 014 and 
015 have been achieved. 
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Figure 9. Survey questions related to the Pedagogical component 


Finally, questions 10, 11 and 16 allow evaluating the perception of the students with regards to the ES activity 
as a whole (see Fig. 10). In particular, question 10 directly asks for the levels of satisfaction, and again more 
than 75% of the students have agreed that the activity has been satisfactory or highly satisfactory, whereas only 
a 6% disagreed (no one strongly disagreed) and the rest remained neutral. Questions 11 and 16 illustrate that 
students demand for extra activities similar to the ones presented in this paper. Specifically, a 79% agreed or 
strongly agreed to question 11 and an 82% agreed or strongly agreed to question 12. To be remarked that the 
69% of the students strongly agreed with the idea that this kind of Educational Simulation activities should be 
also introduced in other subjects of the degree (only a 5% disagree to such idea and no one strongly disagreed). 
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Figure 10. Survey questions related to the Educational Simulation activity as a whole 


4 CONCLUSIONS 

A Learning-by-Doing teaching experience that has made use of an Educational Simulation system has been 
performed and presented in this paper. Specifically, a basic flight simulation environment has been set up, 
which has allowed the students experimenting and conducting some of the most common tasks of an aircraft 
pilot. The activity has been designed for students of the degree on Aeronautical Management in order to 
improve their competencies in the field of the air transportation, specifically providing them with a clearer 
picture about the most important processes that are involved in the commanding and navigation of an aircraft 
(including the pilot's logical reasoning), as well as increasing their ability to understand and detect potential 
areas of improvement based on the use of the new technologies and higher levels of automation. Such 
capabilities may contribute in turn to improve the decision-making competencies of the students in their future 
professional working positions related to the air transportation market. 

The acquisition by part of the students of some of the pursued competencies (knowledge, skills and attitudes 
related to aircraft commanding and navigation) has been successfully observed and assessed during the 
execution of the activities. In addition, after the execution of the teaching experience it has been evaluated with 
an objective test and an attitude and opinion survey, both filled by 38 students six weeks later and without any 
previous announcement (i.e., surprise exam). After analysing the results of the different assessment methods 
used (i.e., objective test, execution test, observation techniques and attitude survey) it can be argued that the 
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teaching experience has been a success in terms of the objectives pursued. 

The results have indicated that the students have learned and assimilated the pursued concepts of flight 
commanding and navigation with a very high success rate, i.e., the 92,11% of students have passed the surprise 
exam while all the students succeed in the execution test and in the evaluation of attitudes. Furthermore, 
according to the results of the opinion survey, it can be stated that the acquisition of the software and 
hardware products used for the flight simulation experience has been a wise decision. Finally, the levels of 
satisfaction with regards to the Educational Simulation experience have been also very high, showing that most 
of the students have enjoyed the experience while they admit that these activities have contributed to their 
learning process; congruently, they express a demand for extra similar Educational Simulation activities as part 
of their university studies. 

As seen, Educational Simulation can provide to teachers with an interesting framework to perform highly 
effective learning experiences, in particular enabling the development of Learning-by-Doing activities that can 
facilitate the learning process of the students and also the acquisition of competencies that otherwise might be 
difficult to achieve. 
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